test(otel): cover replay and chained execution - #796

Add OpenTelemetry conformance handlers for user-function and callback tracing contexts (22–23), invocation retries reported as RETRYING/UNSET (24), caught callback failure replay (25), external callback completion and later replay (26), and a chained child that suspends and resumes through a durable wait (27). These handlers exercise the public Core 3 / OTel 2 APIs in both telemetry views.

Case 21 retains its handler and local replay regression tests, but cloud coverage is explicitly NOT_IMPLEMENTED until the shared history requirement accepts both valid wait-completion orderings. Cases 25–26 are enabled in both cloud suites and the ordinary local test suite. Keep their failures as merge gates until the prerequisite bug fixes merge; do not skip, xfail, or suppress them. The existing case 17 handler remains active; its additional callback-error regression is deferred.

The case 11 callback/replay fix and regression test have merged in #864 and are absent from this PR's diff against main.

Review decisions and simpler alternatives

• Case 21 history ordering: Both current shared requirements assign event 5 to InvocationCompleted and event 6 to WaitSucceeded. The execution-view cloud run observed WaitSucceeded 63 ms earlier while the execution succeeded with before-after. The wait timer and invocation cleanup can finish in either order. Follow the review's deferral option: remove both cloud functions and their build targets and declare both cases pending. The invocation catalog imposes the same ordering, so deferring only the execution view leaves the same risk. Increasing the wait merely reduces the race; it does not establish an ordering guarantee. Re-enable cloud case 21 after a coordinated shared-requirement correction preserves the replay assertions while accepting both orders. Local tests still require each step body to run once, PENDING/SUCCEEDED invocations, and one export per completed operation in the execution view.
• Case 23 error checks: Retain rejection of an unrelated helper failure and of unexpected success. Silently accepting either outcome can hide a failed probe or skip the retry-strategy callback that this scenario must exercise. Extend the existing public-runner test with both outcomes in both views, keeping the original successful-scenario parent/count assertions intact. Each negative case asserts the failure message and that execution does not continue into the virtual child. Removing either check makes its two tests fail.
• Case 22 test harness: The existing OTel branch-isolation tests cover context variables, baggage, and replay in their own handlers; the invocation-resume tests cover handler/step spans. Neither verifies every parent and raw export count from the actual case 22 handler deployed by this PR. Reusing only those tests would miss changes to that handler's child, parallel, and map callbacks. Running case 22 without synchronization could serialize siblings and miss a context leak. Keep four test-only barriers around the existing probes to force overlap at sibling bodies and nested steps; they do not inject or repair tracing contexts. The entry wrapper supplies unrelated ambient host instrumentation and observes restoration on the actual invocation threads. Moving this synchronization into deployed handlers or adding a shared harness would add unnecessary runtime or cross-test coupling.

Workflow and prerequisite fixes

The shared OpenTelemetry workflow and test catalog default to the conformance repository's main branch. Manual runs retain the conformance_test_ref override; SDK handlers run from the commit under test. The orchestrator runs invocation and execution suites. Legacy long-running templates and local contract tests remain, but the orchestrator no longer schedules that suite.

Cases 25 and 26 are registered in the SAM template and Makefile, and their local tests are collected by the existing root test configuration and hatch run test:cov. The prerequisite issues remain open:

• Case 25 requires terminal failure status #853 and empty-error history/runtime consistency #862.
• Case 26 requires completion telemetry #854, callback update production #860, and input acknowledgment #861.
• The additional case 17 callback-error test requires snapshot isolation #852.

Case 25 asserts the intended local-runner contract in #862: detailed callback-failure history includes Payload: {} with Truncated: false; metadata-only history omits the payload and sets Truncated: true. This exact empty Error.Payload service shape remains unverified against deployed AWS Lambda. The restored test inspects raw spans and records the invocation when each span is exported; timestamp-only checks cannot establish delivery timing. It uses ordinary pytest diagnostics, with no configurable proof-file output or JSON reporting mechanism.

Case 26 asserts inside the observed step that completion has already reached the exporter, and preserves raw export-count checks at both later barriers and final completion.

The additional case 17 regression remains deferred. Its source is preserved in 6484e577; before restoring it, correct its history expectations so included diagnostic payloads are untruncated and metadata-only history neither exposes payloads nor reconstructs error details.

Validation

On Python 3.13.17:

• hatch run test:all packages/aws-durable-execution-sdk-python-conformance-tests-otel/tests .github/scripts/tests/test_opentelemetry_conformance_workflow.py -q --tb=short: 4 failed, 89 passed. The failures are intentional regression gates against the current base: case 25 in both views is missing the empty detailed-history payload ([Testing]: Preserve empty callback error history without changing runtime errors #862); case 26 in both views observes zero terminal exports inside the observed step, causing an extra retry and shifted history gates ([OTel]: Export unvisited replay completions with the correct parent context #854/[Testing]: Record callback terminal completions as updated operations #860/[Testing]: Acknowledge invocation input without losing pending updates #861). Later assertions remain enabled, including case 25's terminal ERROR status checks for [OTel]: Report terminal failures and keep exception events within span bounds #853.
• Removing case 23's unexpected-error check produces 2 failures; removing its unexpected-success check produces 2 failures. Both checks are restored in the final source.
• Package formatting/lint, commit-message validation, and git diff --check: passed.
• All 15 caller inputs and 7 secrets were validated against the shared workflow's current schema during the main-branch merge.
• Cloud conformance for this revision is pending. CI is expected to remain failing until the prerequisite behavior is fixed; there are no new failure-suppression settings.

Existing test expectation changes

• test_templates_map_every_otel_requirement, test_templates_declare_only_the_expected_functions, test_function_names_identify_the_requirement, test_makefile_builds_every_function, test_every_registered_handler_exists, and test_chained_invoke_functions_point_at_their_target: include cloud cases 22–27 and case 27's targets; case 21 stays out of the cloud mappings/build targets while its shared history contract is pending.
• test_active_callback_and_retry_handlers_keep_their_mapping: verifies that cases 25 and 26 use their restored replay handlers, alongside the existing case 17 and 24 mappings.
• test_handler_modules_are_exactly_the_expected_set: includes the new active handlers plus case 21's retained local handler.
• test_deferred_requirements_are_explicit: only case 21 remains pending in both views, with a reason and complete accounting for all 27 cases. Cases 25 and 26 are active requirements.
• test_callback_probes_keep_their_actual_sdk_parent: adds unexpected helper failure/success outcomes to protect both error checks; the original successful-scenario assertions are unchanged.
• test_opentelemetry_conformance_caller_uses_current_workflow_contract: expects the shared workflow and catalog to default to main.

Case 27 companion PRs: JS #1068 and Java #824.